Linux compute resource within PFR, first in first out queuing system

Open lava scheduler is used by staff to allocate computational jobs to the available resource.

## Number of available processors $ bhosts | perl -lane '$a+=$F[3]; END{print $a}' 756

The scheduler allocates and records computer resources for each submitted job. When submitting any job to the scheduler a user should know in advance approximately what resource is required and more importantly how long the job should take

Scheduled jobs can be classified into three categories Pending jobs which were cancelled before running. Run jobs with an error status, a failed job. Run jobs with no error status, a successful job. Summarizing the analysis of 5 years of scheduler usage logs A third of the cpu time is wasted Many users are submitting inefficient jobs with excessive run times.

Usually workflows using uncompressed data causes disk thrashing

Perl script to process 5 years of open lava scheduler logs ( lsb.acct file) and write to an SQLite database

knitr::kable(head(cpu_table, n=3))

• CPU time is the amount of central processing time taken (user time + system time)
• Wallclock time is the actual time taken to run
• Percent = 100 x cpu_time / wallclock
• Pending time is the amount of time between submitting a job and it starting to run.

knitr::kable(head(cpu_table %>% select(-command), n=1))

If CPU time is much less than wallclock time there could be a problem

Two issues;

• Excessive memory usage
• Excessive I/O - data transfer between the hard disk drive and RAM

Perl script to dynamically process active directory information

With knowledge about Users and Teams we can further categorize usage

knitr::kable(teams                           %>%
  filter(team_name == "Bioinformatics Team") %>% 
  head(n=3))

Team contains 8/16 errors

"Ashley Lu", "Helge Dzierzon" have left, "Haipeng Zhang", "Jian Guo" are visitors, "Shakira Johnson" not in the team. "Ali Saei", "Marcus Davy", "Simon Deroles" are missing

## Add Bioinformatics
teams$group <- bioinf_team(teams, power_users = TRUE)
teams$group <- shinyClimateChange_team(teams, group = teams$group, power_users = TRUE)
knitr::kable(head(teams))

All run jobs;

There are 2556473 successfully executed jobs over 5.96 years.

All failed jobs;

All successful jobs;

28.3% jobs failed

Summarizing job execution durations

totals <- sum_cpu_time(cpu %>% filter(cpu_time>0))
knitr::kable(totals)

The total CPU time used by all users is 11608914676.7243s (~367.86 years).

This breaks down to 8782821888.15199s (~278.31 years) of successful jobs and 2826092788.57458s (~89.55 years) of failed jobs.

The goal is zero years of failed jobs

Can summarize by username or team

per_user <- summarise_user_usage(cpu)
knitr::kable(head(anonymize(per_user)))

Problem when pending time is greater than run time

Coloured by Username → behavioural problem

Need to fully automate scheduler log analysis into accessible reports.

Urgent requirement for

• Documentation - (how to use the scheduler)
• Staff training - (troubleshooting, and what not to do)

within PFR using Powerplant compute resources

github:powerPlant/03_Openlava.Rmd